[Re-issue: Fix] Keys and Teams Router Setting + Allow Override of Router Settings by yuneng-jiang · Pull Request #20205 · BerriAI/litellm

yuneng-jiang · 2026-02-01T00:08:12Z

Relevant issues

Fixes #19693

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
My PR passes all unit tests on make test-unit
My PR's scope is as isolated as possible, it only solves 1 specific problem

CI (LiteLLM team)

CI status guideline:

50-55 passing tests: main is stable with minor issues.

45-49 passing tests: acceptable but needs attention

<= 40 passing tests: unstable; be careful with your merges and assess the risk.

Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:

Type

🐛 Bug Fix
✅ Test

Changes

This PR fixes search tools not being found when using per-request routers by:

Avoiding per-request Router instantiation: Instead of creating a new Router instance when a key/team has router_settings (which is expensive and loses search_tools), settings are now passed as router_settings_override kwargs to the main router.
Creating router even with empty model list: The router is now created even when model_list is empty if search_tools are configured, since the router can function with only search tools.
Supporting per-request model_group_retry_policy: Added support for model_group_retry_policy as a per-request override in the Router's retry logic.
Using cached key/team objects for router settings: _get_hierarchical_router_settings reads from the already-authenticated key object and the cached team lookup (get_team_object) instead of making direct DB queries on the hot path.

The affected settings that can be overridden per-request are: fallbacks, context_window_fallbacks, content_policy_fallbacks, num_retries, timeout, and model_group_retry_policy.

Added router_settings field to LiteLLM_VerificationToken to support key-level router settings.

Screenshots

Testing

Testing was done mainly with fallbacks with these 3 mappings:
Key: claude-haiku-4-5 will fallback to gpt-4o
Team: claude-haiku-4-5 will fallback to claude-sonnet-4-5
Global: claude-haiku-4-5 will fallback to grok-4-fast-non-reasoning

Team only fallback (key without router settings):

Key and team with router settings:

**Root Cause:** When API keys or teams have router_settings configured, the proxy creates per-request Router instances from user_config. These new routers were missing search_tools from the main router, causing "search tool not found" errors despite search_tools being configured. **The Fix:** 1. **common_request_processing.py (lines 554-556):** Pass search_tools from main router to user_config so per-request routers inherit them 2. **proxy_server.py (line 3171):** Remove `if len(_model_list) > 0` check to allow router creation with empty model list (needed for search-tools-only use case) 3. **proxy_server.py (line 746):** Remove redundant search_tools loading code (already handled by _init_search_tools_in_db() called during startup)

fix: search tools not found when using per-request routers

vercel · 2026-02-01T00:08:17Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 6, 2026 11:09pm

greptile-apps · 2026-02-01T00:11:17Z

Greptile Overview

Greptile Summary

This PR fixes a critical performance issue where search_tools were not accessible when using per-request router settings. The fix replaces expensive per-request Router instantiation with a router_settings_override approach that merges settings into the main router's kwargs.

Key Changes:

Stores hierarchical router settings as router_settings_override instead of creating a new Router with user_config
Allows Router creation even with empty model_list when search_tools are configured
Adds per-request model_group_retry_policy override support in the Router's retry logic
Comprehensive test coverage validates the new approach and ensures request-level settings take precedence over key/team settings

Performance Impact:
This change directly addresses the custom rule about avoiding Router object creation in the critical request path, significantly improving request latency by reusing the main router instance.

Confidence Score: 5/5

This PR is safe to merge with minimal risk
The changes are well-architected, follow best practices, and include comprehensive test coverage. The approach correctly avoids expensive Router instantiation per-request while maintaining the same functionality. All settings precedence is properly handled (request-level > key/team > global).
No files require special attention

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Replaced per-request Router instantiation with router_settings_override to avoid expensive object creation, following performance guidelines
litellm/proxy/route_llm_request.py	Replaced user_config Router instantiation with router_settings_override merging, respecting request-level precedence
litellm/router.py	Added per-request model_group_retry_policy override support and updated retry logic to use helper function

Sequence Diagram

sequenceDiagram
    participant Client
    participant ProxyServer
    participant CommonRequestProcessing
    participant RouteRequest
    participant Router
    participant LLMProvider

    Client->>ProxyServer: POST /chat/completions
    ProxyServer->>CommonRequestProcessing: process_request()
    
    Note over CommonRequestProcessing: Get hierarchical router settings<br/>(from key/team/global config)
    
    alt Router settings found
        CommonRequestProcessing->>CommonRequestProcessing: Store as router_settings_override
        Note over CommonRequestProcessing: OLD: Created new Router instance<br/>NEW: Store settings for later merge
    end
    
    CommonRequestProcessing->>RouteRequest: route_request(data, router, ...)
    
    alt router_settings_override exists
        RouteRequest->>RouteRequest: Extract override_settings
        loop For each per_request_setting
            RouteRequest->>RouteRequest: Merge into data if not already set
        end
        Note over RouteRequest: Settings: fallbacks, num_retries,<br/>timeout, model_group_retry_policy, etc.
        RouteRequest->>Router: acompletion(**data)
        Note over Router: Uses main router with<br/>merged override settings
    else No override
        RouteRequest->>Router: acompletion(**data)
    end
    
    Router->>Router: async_function_with_retries()
    Note over Router: Extract per-request overrides:<br/>- fallbacks<br/>- model_group_retry_policy<br/>- timeout, num_retries
    
    Router->>LLMProvider: Make API call
    
    alt Success
        LLMProvider-->>Router: Response
        Router-->>RouteRequest: Response
        RouteRequest-->>ProxyServer: Response
        ProxyServer-->>Client: 200 OK
    else Error with retry policy
        LLMProvider-->>Router: Error
        Router->>Router: _get_num_retries_from_retry_policy()
        Note over Router: Uses per-request<br/>model_group_retry_policy<br/>if provided
        Router->>LLMProvider: Retry with fallback
        LLMProvider-->>Router: Response
        Router-->>RouteRequest: Response
        RouteRequest-->>ProxyServer: Response
        ProxyServer-->>Client: 200 OK
    end

greptile-apps

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

yuneng-jiang · 2026-02-06T23:08:04Z

@greptile

greptile-apps · 2026-02-06T23:10:59Z

Greptile Overview

Greptile Summary

This PR optimizes router settings handling to avoid expensive per-request Router instantiation. Previously, when keys or teams had custom router settings, a new Router object was created for each request, which was costly and lost search_tools configuration.

Key changes:

Performance improvement: Router settings from keys/teams are now passed as router_settings_override kwargs to the main router instead of creating new Router instances, adhering to the custom rule about avoiding object creation in the critical request path
Cached lookups: Uses already-cached key objects and team cache (get_team_object) to retrieve hierarchical settings (key > team > global) without direct DB queries
Search tools support: Router is now created even with empty model_list when search_tools are configured, fixing the search tools not found issue
Per-request retry policy: Added support for model_group_retry_policy as a per-request override in the Router's retry logic

The implementation correctly merges only supported per-request settings (fallbacks, context_window_fallbacks, content_policy_fallbacks, num_retries, timeout, model_group_retry_policy) and respects request-level precedence over key/team settings.

Confidence Score: 4/5

This PR is safe to merge with minimal risk after addressing the exception handling issue
The PR implements a well-tested performance optimization that aligns with the custom rule about avoiding object creation in the request path. The approach is sound: using cached lookups and passing settings as kwargs instead of creating new Router instances. Tests are comprehensive and cover edge cases. One minor syntax issue exists in exception handling (line 3426) where yaml.safe_load() can't raise json.JSONDecodeError, but this won't cause failures - it just makes the except clause unnecessarily broad. The change is backwards compatible and follows existing patterns in the codebase.
Pay attention to litellm/proxy/proxy_server.py for the exception handling fix on line 3426

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Retrieves hierarchical router settings (key > team) and passes them as `router_settings_override` instead of creating new Router
litellm/proxy/proxy_server.py	Added `_parse_router_settings_value` and `_get_hierarchical_router_settings` methods, removed empty model_list check for Router creation
litellm/proxy/route_llm_request.py	Replaced expensive per-request Router instantiation with settings merge into kwargs for main router
litellm/router.py	Added per-request `model_group_retry_policy` override support in retry logic using existing helper function

Sequence Diagram

sequenceDiagram
    participant Client
    participant ProxyBaseLLMRequestProcessing
    participant ProxyConfig
    participant UserAPIKeyCache
    participant TeamCache
    participant RouteRequest
    participant Router
    
    Client->>ProxyBaseLLMRequestProcessing: Request with API Key
    ProxyBaseLLMRequestProcessing->>ProxyConfig: _get_hierarchical_router_settings()
    ProxyConfig->>UserAPIKeyCache: Get key object (cached)
    UserAPIKeyCache-->>ProxyConfig: user_api_key_dict with router_settings
    
    alt Key has router_settings
        ProxyConfig-->>ProxyBaseLLMRequestProcessing: Return key-level settings
    else No key settings, check team
        ProxyConfig->>TeamCache: get_team_object() (cached)
        TeamCache-->>ProxyConfig: team object with router_settings
        ProxyConfig-->>ProxyBaseLLMRequestProcessing: Return team-level settings
    end
    
    ProxyBaseLLMRequestProcessing->>ProxyBaseLLMRequestProcessing: Add router_settings_override to data
    ProxyBaseLLMRequestProcessing->>RouteRequest: route_request(data, router)
    
    RouteRequest->>RouteRequest: Extract router_settings_override
    RouteRequest->>RouteRequest: Merge allowed settings into data kwargs
    Note over RouteRequest: Only merge: fallbacks, context_window_fallbacks,<br/>content_policy_fallbacks, num_retries,<br/>timeout, model_group_retry_policy
    
    RouteRequest->>Router: router.acompletion(**data)
    Note over Router: Uses main Router with<br/>per-request overrides<br/>(no new Router created)
    Router-->>RouteRequest: Response
    RouteRequest-->>Client: Response

greptile-apps

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-02-06T23:11:02Z

litellm/proxy/proxy_server.py

+
+            try:
+                parsed = yaml.safe_load(value)
+            except (yaml.YAMLError, json.JSONDecodeError):


yaml.safe_load() doesn't raise JSONDecodeError, only YAMLError. The json.JSONDecodeError in the except clause won't catch yaml parsing failures.

Suggested change

except (yaml.YAMLError, json.JSONDecodeError):

except yaml.YAMLError:

Previously, PR #19818 (via #20205) removed the model_list check entirely, causing Router to be created even with no models AND no search_tools. This fix adds back a conditional check that creates the Router only when: - There are models to route, OR - There are search_tools configured This preserves the PR #19818 goal (search-tools-only deployments) while avoiding unnecessary Router creation when there's nothing to route. Fixes test_add_and_delete_deployments[0-None]

) Previously, PR #19818 (via #20205) removed the model_list check entirely, causing Router to be created even with no models AND no search_tools. This fix adds back a conditional check that creates the Router only when: - There are models to route, OR - There are search_tools configured This preserves the PR #19818 goal (search-tools-only deployments) while avoiding unnecessary Router creation when there's nothing to route. Fixes test_add_and_delete_deployments[0-None]

Quentin-M and others added 3 commits January 26, 2026 17:57

Merge pull request #19818 from Quentin-M/search_tools_fix

93383d0

fix: search tools not found when using per-request routers

Override router settings

a9eae59

vercel bot had a problem deploying to Preview February 1, 2026 00:10 Failure

yuneng-jiang changed the title ~~Litellm router search fix~~ [Fix] Allow Override of Router Settings Feb 1, 2026

greptile-apps bot reviewed Feb 1, 2026

View reviewed changes

yuneng-jiang added 2 commits February 6, 2026 14:08

Merge remote-tracking branch 'origin' into litellm_router_search_fix

400e560

use cached keys and teams for router settings

fd3ca08

vercel bot deployed to Preview February 6, 2026 23:09 View deployment

yuneng-jiang changed the title ~~[Fix] Allow Override of Router Settings~~ [Re-issue: Fix] Keys and Teams Router Setting + Allow Override of Router Settings Feb 6, 2026

greptile-apps bot reviewed Feb 6, 2026

View reviewed changes

yuneng-jiang merged commit 218373c into main Feb 6, 2026
56 of 66 checks passed

shin-bot-litellm mentioned this pull request Feb 7, 2026

fix(proxy): only create Router when models or search_tools exist #20661

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Re-issue: Fix] Keys and Teams Router Setting + Allow Override of Router Settings#20205

[Re-issue: Fix] Keys and Teams Router Setting + Allow Override of Router Settings#20205
yuneng-jiang merged 5 commits intomainfrom
litellm_router_search_fix

yuneng-jiang commented Feb 1, 2026 •

edited

Loading

Uh oh!

vercel bot commented Feb 1, 2026 •

edited

Loading

Uh oh!

greptile-apps bot commented Feb 1, 2026

Important Files Changed

Uh oh!

greptile-apps bot left a comment

Uh oh!

yuneng-jiang commented Feb 6, 2026

Uh oh!

greptile-apps bot commented Feb 6, 2026

Important Files Changed

Uh oh!

greptile-apps bot left a comment

Uh oh!

greptile-apps bot Feb 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	except (yaml.YAMLError, json.JSONDecodeError):
	except yaml.YAMLError:

Uh oh!

Conversation

yuneng-jiang commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Pre-Submission checklist

CI (LiteLLM team)

Type

Changes

Screenshots

Testing

Uh oh!

vercel bot commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps bot commented Feb 1, 2026

Greptile Overview

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

yuneng-jiang commented Feb 6, 2026

Uh oh!

greptile-apps bot commented Feb 6, 2026

Greptile Overview

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yuneng-jiang commented Feb 1, 2026 •

edited

Loading

vercel bot commented Feb 1, 2026 •

edited

Loading